Summary of the research paper analysis template (spelled out in more detail below)
(i) Establishing the context and purpose for the study
1. Contextualizing information about authors, study, etc.
2. Research question and problem area
3. Purpose
(ii) Framing the study
4. Theoretical framework
5. Conceptual framework
(iii) Methodology
6. Research design
• Participant selection and characteristics
• Time frames
• Data collection tools and methods
• Data analysis approach
(iv) Interpretation
7. Data interpretation
• Match between interpretations and the actual data
• Match between interpretations and the framing of the study
8. The paper concludes with a satisfying list of implications and recommendations etc.
9. Overall general questions concerning quality of writing and structure of paper
An analytic template for critically evaluating research report articles
Overall, developing an evaluation of a particular target study may require us to bring additional research texts to bear on our analysis (e.g., in order to comment on a study focused on the “digital divide” in Canada, you may need to follow up on other studies that focus on the digital divide in order to obtain sufficient breadth of analysis; or follow up on previous studies by the same authors, etc.)
(i) Establishing the context and purpose for the study
1. Contextualizing information
a. Who are the researchers? Do they “come clean” on affiliations and vested interests that have direct bearing on the study (e.g., are they testing a software program they stand to profit from?)
b. Who funded the study?
c. What is the scope of the study—do they claim global scope but only conduct their investigation within one country?
d. Where was it published or originally presented?
e. Who appears to be the intended audience, and how do we know?
f. Why is this research being done, and what is it trying to do?
2. Research question
• Is there an obvious, explicitly stated research question (or questions)?
• Does the question match the purpose of the study?
• Are the who, what, where, when (maybe) etc. indicated?
• To what extent does the question signal key concepts to be used in the study?
• Is the question worth asking? Is it relevant to current contexts or to current state of theorizing things?
• Is it manageable and do-able? How do I know?
3. Purpose
• Is the purpose clearly and concisely identified?
• Is it worth addressing? Has this focus already be “done to death”? Does it address a “gap” in knowledge or contribute to knowledge in some fresh way?
• Does the statement of purpose provide a rationale for the project? Is there an intended beneficiary for this purpose? (i.e., who will benefit from this study?)
• Is the purpose grounded in a noteworthy/relevant problem area?
• May include a statement of goals for the study overall
(ii) Framing the study
4. Theoretical framework
• Do the researchers state their theoretical orientation clearly and concisely? If not, why not? Does the framework clearly “play itself out” throughout the report?
• Some researchers may be developing theory out of their findings (e.g., Grounded Theory, which makes use of open coding), or others may be applying theory (e.g., constructivism, sociocultural theory, psychological theory, feminist theory, critical theory, post-structuralist theory, cognitive theory, metacognitive theory, linguistic theory, Actor Network Theory, New Literacy Studies theory, sociological theory, deconstructionist theory, natural learning theory, Whole Language theory, etc.)
• Does the theory “match” the research question?
• Who are they citing in support of their theoretical framework? Who are they aligning themselves with?
• What assumptions about the world are the authors making? These are the kinds of things they are taking for granted, as not needing to be proved.
• How well-informed is the study with respect to drawing on relevant bodies of existing research etc.?
5. Conceptual framework
• Are key concepts easy to identify in the research report? (e.g., “digital divide”, “communication”, “access”, “out-of-school”, the “everyday”)
• Are key concepts explicitly defined? What meaning is given to them?
• Are the concepts used consistently throughout the paper?
• Are key concepts consonant with the theoretical framing set for the study? With each other?
• Are the key concepts used in this study in touch with current times? Are they in keeping with the practices documented/reported in the study? Are they usefully explanatory?
(iii) Methodology
6. Research design
• Is the design clearly stated?
o Is it logical? Does it “match” the purpose and research question?
o Is the method used to select participants discussed explicitly? Is participant selection justified? Are participant characteristics clearly identified? What criteria did participants need to meet in order to be eligible for selection? Are incentives disclosed, and what effect might these have on the study discussed?
o Are the tools and methods used to collect data described sufficiently?
o Is the data collection procedure or process is clearly described?
o Is the time frame is identified and does it match the stated purpose of the study? Is the time frame justified
Research Design
• Do the authors clearly identify what type of study they are conducting, and why? (e.g., a quasi-experimental study that focuses on some kind of intervention, a qualitative study examining a group of students engagement with popular culture in a range of contexts)
• Is the context for the study justified? (i.e., where physically were data collected and is this justified, given the research question etc.?)
• Is there enough information given about the logistics of the study that I could replicate this study or conduct a similar study in a different context etc.?
• In quantitative research, are key “rules” followed with respect to control of variables etc.
• Is this study ethical? How do I know?
Data collection
• Are data collection tools and methods described in reasonable detail and justified with respect to the purpose of the study?
• To what extent are data collection tools and methods consonant with the theoretical and conceptual framework for the study?
• Are sample items etc. of data collection tools and methods provided for the reader to help with judging the quality of the tools and methods?
• Were certain tools, like surveys and logs etc. trialled in order to better ensure validity of outcomes?
• Are the timeframes applicable to data collection tools and methods made explicit? In what way does the timeframe for the study impact on the quality of data obtained?
• Do the tools and methods used enable the researchers to cross-check the information they gather? (i.e., are forms of triangulation built into the study?)
• Am I able to judge whether or not sufficient data has been collected to support the interpretations made by the authors? (This especially holds in qualitative research designs)
Data analysis
• Is the analysis going to stand up to scrutiny?
• Do they specify a specific data analysis method, and does it match their theoretical framing and purpose etc.?
• Are they able to justify the methods used and the results they generated?
• Does the data analysis pay attention to the conceptual framing of the study? (You would expect to see the conceptual framing being revisited in some way in the analysis of data)
Types of data analysis are written up in the handbook (chapters 13, 14 & 15) which might be a useful place for you to start in understanding different approaches to analyzing data. The role of data analysis is to find patterns and regularities in the data and to make sense of these patterns and regularities. Therefore, you need to ask how did the authors arrive at the patterns that they did.
(iv) Interpretation
7. Data interpretation
• Interpretation is the process of making sense of the patterns I find in your data. It is concerned with addressing the “so what?” of the data analysis results. So, I need to ask, do the authors make useful, logical, coherent sense of what the patterns mean? Are interpretations in keeping with the theoretical and conceptual frameworks for this study?
• Does interpretation pay attention to variables that may impact on and shape outcomes?
• How did the researchers select the samples of data that they present?
• Do the interpretations grow out of the data, or are the conclusions in place prior to the study—and what effects does/might this have?
• How are issues of validity and “trustworthiness” addressed? Do the authors clearly identify how they paid attention to validity and reliability in quantitative research and to communicative validity and trustworthiness in qualitative research? For example, do they include participant checks to add weight to the “appropriateness” of their interpretations?
8. Conclusion
• Do the authors identify the limitations of their study and how these impact outcomes? (e.g., do they acknowledge the limits to which their study can be generalized?)
• To what extent did this study address its research question(s) and purpose?
• What conclusions do the authors reach based on the study and in relation to the research question and purpose etc.?
• Are there any implications drawn from the findings? In what ways are these implications important or relevant for the target audience of readers for this paper? Implications logically follow from a study’s findings.
• Do the authors make recommendations for education? To what extent are these recommendations “realistic” or warranted? Are the recommendations consistent with the implications of the study? (Recommendations can be based on opinions). The relationship between implications and recommendations parallels the relationship between data analysis and interpretation.
• Are there suggestions for further research? If not, what might I suggest could be done?
9. Overall general questions to ask
• Is the report itself logically structured and easy to follow?
• Is the quality of writing accessible and clear?
• Is this research report worth reading?
APPENDIX 1
Summary of the research paper analysis template (spelled out in more detail below)
(i) Establishing the context and purpose for the study
1. Contextualizing information about authors, study, etc.
2. Research question and problem area
3. Purpose
(ii) Framing the study
4. Theoretical framework
5. Conceptual framework
(iii) Methodology
6. Research design
• Participant selection and characteristics
• Time frames
• Data collection tools and methods
• Data analysis approach
(iv) Interpretation
7. Data interpretation
• Match between interpretations and the actual data
• Match between interpretations and the framing of the study
8. The paper concludes with a satisfying list of implications and recommendations etc.
9. Overall general questions concerning quality of writing and structure of paper
An analytic template for critically evaluating research report articles
Overall, developing an evaluation of a particular target study may require us to bring additional research texts to bear on our analysis (e.g., in order to comment on a study focused on the “digital divide” in Canada, you may need to follow up on other studies that focus on the digital divide in order to obtain sufficient breadth of analysis; or follow up on previous studies by the same authors, etc.)
(i) Establishing the context and purpose for the study
1. Contextualizing information
a. Who are the researchers? Do they “come clean” on affiliations and vested interests that have direct bearing on the study (e.g., are they testing a software program they stand to profit from?)
b. Who funded the study?
c. What is the scope of the study—do they claim global scope but only conduct their investigation within one country?
d. Where was it published or originally presented?
e. Who appears to be the intended audience, and how do we know?
f. Why is this research being done, and what is it trying to do?
2. Research question
• Is there an obvious, explicitly stated research question (or questions)?
• Does the question match the purpose of the study?
• Are the who, what, where, when (maybe) etc. indicated?
• To what extent does the question signal key concepts to be used in the study?
• Is the question worth asking? Is it relevant to current contexts or to current state of theorizing things?
• Is it manageable and do-able? How do I know?
3. Purpose
• Is the purpose clearly and concisely identified?
• Is it worth addressing? Has this focus already be “done to death”? Does it address a “gap” in knowledge or contribute to knowledge in some fresh way?
• Does the statement of purpose provide a rationale for the project? Is there an intended beneficiary for this purpose? (i.e., who will benefit from this study?)
• Is the purpose grounded in a noteworthy/relevant problem area?
• May include a statement of goals for the study overall
(ii) Framing the study
4. Theoretical framework
• Do the researchers state their theoretical orientation clearly and concisely? If not, why not? Does the framework clearly “play itself out” throughout the report?
• Some researchers may be developing theory out of their findings (e.g., Grounded Theory, which makes use of open coding), or others may be applying theory (e.g., constructivism, sociocultural theory, psychological theory, feminist theory, critical theory, post-structuralist theory, cognitive theory, metacognitive theory, linguistic theory, Actor Network Theory, New Literacy Studies theory, sociological theory, deconstructionist theory, natural learning theory, Whole Language theory, etc.)
• Does the theory “match” the research question?
• Who are they citing in support of their theoretical framework? Who are they aligning themselves with?
• What assumptions about the world are the authors making? These are the kinds of things they are taking for granted, as not needing to be proved.
• How well-informed is the study with respect to drawing on relevant bodies of existing research etc.?
5. Conceptual framework
• Are key concepts easy to identify in the research report? (e.g., “digital divide”, “communication”, “access”, “out-of-school”, the “everyday”)
• Are key concepts explicitly defined? What meaning is given to them?
• Are the concepts used consistently throughout the paper?
• Are key concepts consonant with the theoretical framing set for the study? With each other?
• Are the key concepts used in this study in touch with current times? Are they in keeping with the practices documented/reported in the study? Are they usefully explanatory?
(iii) Methodology
6. Research design
• Is the design clearly stated?
o Is it logical? Does it “match” the purpose and research question?
o Is the method used to select participants discussed explicitly? Is participant selection justified? Are participant characteristics clearly identified? What criteria did participants need to meet in order to be eligible for selection? Are incentives disclosed, and what effect might these have on the study discussed?
o Are the tools and methods used to collect data described sufficiently?
o Is the data collection procedure or process is clearly described?
o Is the time frame is identified and does it match the stated purpose of the study? Is the time frame justified
Research Design
• Do the authors clearly identify what type of study they are conducting, and why? (e.g., a quasi-experimental study that focuses on some kind of intervention, a qualitative study examining a group of students engagement with popular culture in a range of contexts)
• Is the context for the study justified? (i.e., where physically were data collected and is this justified, given the research question etc.?)
• Is there enough information given about the logistics of the study that I could replicate this study or conduct a similar study in a different context etc.?
• In quantitative research, are key “rules” followed with respect to control of variables etc.
• Is this study ethical? How do I know?
Data collection
• Are data collection tools and methods described in reasonable detail and justified with respect to the purpose of the study?
• To what extent are data collection tools and methods consonant with the theoretical and conceptual framework for the study?
• Are sample items etc. of data collection tools and methods provided for the reader to help with judging the quality of the tools and methods?
• Were certain tools, like surveys and logs etc. trialled in order to better ensure validity of outcomes?
• Are the timeframes applicable to data collection tools and methods made explicit? In what way does the timeframe for the study impact on the quality of data obtained?
• Do the tools and methods used enable the researchers to cross-check the information they gather? (i.e., are forms of triangulation built into the study?)
• Am I able to judge whether or not sufficient data has been collected to support the interpretations made by the authors? (This especially holds in qualitative research designs)
Data analysis
• Is the analysis going to stand up to scrutiny?
• Do they specify a specific data analysis method, and does it match their theoretical framing and purpose etc.?
• Are they able to justify the methods used and the results they generated?
• Does the data analysis pay attention to the conceptual framing of the study? (You would expect to see the conceptual framing being revisited in some way in the analysis of data)
Types of data analysis are written up in the handbook (chapters 13, 14 & 15) which might be a useful place for you to start in understanding different approaches to analyzing data. The role of data analysis is to find patterns and regularities in the data and to make sense of these patterns and regularities. Therefore, you need to ask how did the authors arrive at the patterns that they did.
(iv) Interpretation
7. Data interpretation
• Interpretation is the process of making sense of the patterns I find in your data. It is concerned with addressing the “so what?” of the data analysis results. So, I need to ask, do the authors make useful, logical, coherent sense of what the patterns mean? Are interpretations in keeping with the theoretical and conceptual frameworks for this study?
• Does interpretation pay attention to variables that may impact on and shape outcomes?
• How did the researchers select the samples of data that they present?
• Do the interpretations grow out of the data, or are the conclusions in place prior to the study—and what effects does/might this have?
• How are issues of validity and “trustworthiness” addressed? Do the authors clearly identify how they paid attention to validity and reliability in quantitative research and to communicative validity and trustworthiness in qualitative research? For example, do they include participant checks to add weight to the “appropriateness” of their interpretations?
8. Conclusion
• Do the authors identify the limitations of their study and how these impact outcomes? (e.g., do they acknowledge the limits to which their study can be generalized?)
• To what extent did this study address its research question(s) and purpose?
• What conclusions do the authors reach based on the study and in relation to the research question and purpose etc.?
• Are there any implications drawn from the findings? In what ways are these implications important or relevant for the target audience of readers for this paper? Implications logically follow from a study’s findings.
• Do the authors make recommendations for education? To what extent are these recommendations “realistic” or warranted? Are the recommendations consistent with the implications of the study? (Recommendations can be based on opinions). The relationship between implications and recommendations parallels the relationship between data analysis and interpretation.
• Are there suggestions for further research? If not, what might I suggest could be done?
9. Overall general questions to ask
• Is the report itself logically structured and easy to follow?
• Is the quality of writing accessible and clear?
• Is this research report worth reading?